Methods and Systems for Communicating Using a Virtual Private Network

ABSTRACT

Systems and methods for communication. A network abstraction layer (NAL) is built on a public Internet; and a network virtualization layer (NVL) is built on the NAL.

RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/325,757, filed Jul. 8, 2014, which is a Continuation of U.S. patentapplication Ser. No. 12/471,199 (now U.S. Pat. No. 8,837,491), filed May22, 2009, which claims priority to U.S. Provisional Application No.61/056,268 filed May 27, 2008, which are incorporated herein byreference in their entirety as if fully set forth herein.

This application is also related to U.S. patent application Ser. No.12/471,179 filed May 22, 2009 entitled Global Virtual VPN (nowAbandoned) which is incorporated herein by reference in its entirety asif fully set forth herein.

FIELD OF INVENTION

The present invention relates in general to network communications andspecifically to create improved virtual private networks over theInternet, with unattended provisioning features for network serviceproviders and virtualized physical platforms.

BACKGROUND

A VPN solution is a communication network that connects differentprivate regions through another network. There are two types of VPNs: IPVPNs and IPSec VPNs. An IP VPN is a dedicated network service using aprovider's private network as the transport means. For instance,MPLS-based solutions are IP VPNs. An IPSec VPN is a network thatleverages a public infrastructure like Internet as the transportmechanism. As it runs over a public network, the data is encrypted bythe VPN devices as they exit the regions using ciphering techniques likeIPSec protocol to ensure privacy and man-in-the-middle attacks.

VPNs comprise of two components as shown on FIG. 2: the hubs and thespokes. The hubs have the roles of aggregating and authenticating allthe members connecting to the same VPN network. The spokes are themembers of that VPN network. Spokes encrypt the traffic before sendingit to another member over the public network. When traffic encrypted isreceived from the Internet, the spokes decrypt the traffic and hand itoff to the private networks.

IP VPNs have lots of advantages like strong Service Level Agreements(SLA) or good performance but they are very expensive as well. In theother hand, IPSec VPNs are cheap alternative to these IP VPN solutions.But they are far from providing the same level of service due to thetechnology limitations. They are most of the time based on a networktopology that requires the traffic to always transit via a central pointbefore reaching any destination. Multimedia traffic is not handledeasily as quality of service (QoS) is not supported (because when thetraffic gets encrypted, it can't be classified by QoS capable devicesalong the way and therefore is treated in a best effort manner). Also,IPSec VPNs are using devices that are deployed using a per-customerbasis. They can't be shared between customers. IPSec VPN devices canonly be members of one IPSec VPN network. Finally, Internet-based VPNnetworks also introduce a significant network performance degradationcompared to IP VPNs. This can affect time sensitive applications fromrunning correctly, impacting the user experience, especially in aworldwide deployment.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying figures where like reference numerals refer toidentical or functionally similar elements throughout the separate viewsand which together with the detailed description below are incorporatedin and form part of the specification, serve to further illustratevarious embodiments and to explain various principles and advantages allin accordance with the present invention.

FIG. 1 is a block diagram illustrating a network communication networkin accordance with some embodiments of the invention.

FIG. 2 is a prior art network diagram showing how some communicationsnetwork are setup today.

FIG. 3 is a prior art network diagram showing how some communicationsnetwork are setup today.

FIG. 4 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 4 bis is a network diagram showing further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 4 ter is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 5 is a network diagram showing further steps for building improvedvirtual private networks in accordance with the embodiments of theinvention.

FIG. 6 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 7 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 8 is a network diagram showing further steps for building improvedvirtual private networks in accordance with the embodiments of theinvention.

FIG. 8 bis is a network diagram showing further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 9 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 10 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

FIG. 11 is a network diagram illustrating further steps for buildingimproved virtual private networks in accordance with the embodiments ofthe invention.

Skilled artisans will appreciate that elements in the figures areillustrated for simplifying and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to helpimprove understanding of embodiments of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

Before describing in detail embodiments that are in accordance with thepresent invention, it should be observed that the embodiments resideprimarily in combinations of method steps and apparatus componentsrelated to providing faster Internet-based virtual private networks.Accordingly, the apparatus components and method steps have beenrepresented where appropriate by conventional symbols in the drawings,showing only those specific details that are pertinent to understandingthe embodiments of the present invention so as not to obscure thedisclosure with details that will be readily apparent to those ofordinary skill in the art having the benefit of the description herein.

In this document, the terms “comprises”, “comprising” or any othervariation thereof, are intended to cover a non-exclusive inclusion, suchas a process, method, article or apparatus that comprises a list ofelements does not include only those elements but may include otherelements not expressly listed or inherent such process, method, articleor apparatus. An element proceeded by “comprises . . . a” does not,without more constrains, preclude the existence of additional identicalelements in the process, method, article or apparatus that comprises theelement.

It will be appreciated that embodiments of the invention describedherein may be comprised of one or more conventional network devices orendpoints and unique stored configurations that control the one of morenetwork devices to implement, in conjunction with certain networkcircuits, some, most, or all of the functions of method and apparatusfor providing improved virtual private networks described herein. Thenetwork devices may include, but are not limited to, a centralprocessing unit (CPU), volatile and non-volatile memory banks, networkinterface cards or ports, power source circuits and user input devices.As such, these functions may be interpreted as steps of a method thatdelivers improved virtual private networks. Alternatively, some or allthe functions could be implemented by a state machine that has no storedprogram instructions, or in one or more application specific integratedcircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic. Of course, acombination of the two approaches could be used. Thus, methods and meansfor these functions have been described herein. Further, it is expectedthat one of ordinary skill, notwithstanding possibly significant effortand many design choices motivated by, for example, available time,current technology, and economic considerations, when guided by theconcepts and principles disclosed herein will readily capable ofgenerating such software instructions and programs in minimal, and notundue, experimentation.

An embodiment of the invention is to improve the communication betweentwo routing devices located in different continent. According to oneembodiment, one enhancement includes the attachment of the routingdevices to a closest point of presence within the same continent toovercome the unpredictable behavior of Internet between continents.Another embodiment of the invention is the unattended IP routinginformation distributed over the Internet by daemons to all theendpoints using host-based static routing information only (no defaultgateway used in endpoint configurations). According to one embodiment,using shared encryption keys between endpoints of the same networksolves the resource management (memory, CPU) of the endpoints inaddition to improving the network responsiveness. Another embodiment ofthe invention is to improve the X.509 certificates delivery process andassociated services by using load-balanced certification authorities.According to the same embodiment, the resulting network design alsoclaims a better network protection of the certification authorities.Another embodiment of the invention is to improve the number ofdifferentiated networks on the same aggregating devices located at thepoints of presence. According to one embodiment, the use ofvirtualization capabilities of the routing devices may allow the trafficfrom different origins to be handled by the same physical devices. Thearchitecture of the physical platforms and the logical network topologythat enable the Virtual VPN solution constitute other embodiment.According to another embodiment, advanced traffic scheduling techniquesare used to manage the behavior of the network packets over the lastmile (i.e. the circuit connected to the endpoint). According to anotherembodiment, the endpoint interface scheduling behavior is optimized byreducing the transmit ring queue length. According to anotherembodiment, the particular network topology enabled the use of a fullyautomated unattended remote provisioning methodology.

As shown on FIG. 1, two layers are built on top of the Internet (001) aspart of the Virtual VPN solution: the Network Abstraction Layer (NAL)(002) and the Network Virtualization Layer (NVL) (003). The NAL (002)creates a network underlying foundation to support the NVL (003).

The NAL (002) relies on Generic Routing Encapsulation (GRE) protocol.GRE is a tunneling protocol designed to encapsulate a wide variety ofnetwork layer packets inside IP tunneling packets. A tunneling protocolis used when one network protocol called the payload protocol isencapsulated within a different delivery protocol. GRE tunnelingprotocol is used to provide a cloud of virtual paths, the NAL (002)through an untrusted network (001). As shown on FIG. 2, a NAL (002)consist of a GRE network (011) built over the Internet (001). Theendpoints of the NAL (002) are devices (012, 012, 013, 015) that canhave connectivity between each other. One endpoint, called the hub(012), is the network intelligence of the GRE network (011). It isresponsible for registration of the other endpoints (013, 014, 015) inthe network. When a data flow goes from a hub (012) to or from a spoke(013, 014 or 015), the type of traffic flow is called hub-to-spoke(017). This refers to a point-to-point GRE network (p2p GRE). When adata flow goes from a spoke (013, 014 or 015) to another spoke (013, 014or 015), the type of traffic flow is called spoke-to-spoke (016). Thisrefers to a point-to-multipoint GRE network or multipoint GRE network(mGRE). mGRE networks have all the specifications of the p2p GRE networkwith the improvement of a spoke to be directly capable of reachinganother spoke without the traffic to transit via a hub (assuming theunderlying network on which the GRE network is built is capable ofrouting from one spoke to another; that is the case for the Internetused as the underlying network). In the case of this Virtual VPNsolution, multipoint GRE (mGRE) over the Internet is used to create theNetwork Abstraction Layer (NAL). mGRE creates virtual links between allthe endpoints (routers) over the public network. mGRE networks compriseof two types of nodes: hubs (also called head-ends) and spokes. Hubs arerouters that have the role of aggregation and are in charge ofmaintaining a database with all the information of the spokes within themGRE network or cloud. Usually the hubs are hosted in a data centre. Thespokes are routers that are members of the cloud. The spokes are at thelocations that need to communicate to all the other location or/and tothe date centre. An mGRE network includes, but is not limited to, onehub. Several hubs can be deployed for resiliency.

As shown on FIG. 3 as a prior art drawing, an mGRE network in worldwidedeployment can have an impact on the user experience as the hub-to-spokeor spoke-to-spoke traffic relies on the Internet routing capabilities.As illustrated on FIG. 3, a standard spoke-to-spoke mGRE network withendpoints in America (020), Europe (021) and Asia (022) continents.There are one spoke (026) in Sweden, one spoke (024) in Spain, one spoke(025) in India, one spoke in the US (035) and one spoke (034) in Canada.All of them connect with virtual links (027, 028, 029) to a hub (023)located in the US. These are virtual links because there are linksestablished over the Internet: there is no physical links between theselocations; the traffic is routed from one location to another via thepublic network. When a spoke needs to reach another spoke, directvirtual links from one spoke to another spoke are dynamicallyestablished on demand, by establishing an encrypted tunnel over theInternet. If the spoke (026) located in Sweden needs to establish acommunication with the spoke (024) located in Spain, a spoke-to-spokelink is built (032). The exact same scenario happens when a spoke (026)located in Sweden has to talk with a spoke (025) located in India, aspoke-to-spoke link is built (036). The issue in this scenario is thatthe transit between Asia and Europe has an unpredictable behavior due tothe distance: packet loss, latency or long round trip time (RTT) as mostof the traffic between Europe and Asia mostly passes through the US.Some applications are very sensitive to latency. Latency is a time delaybetween the moment something is initiated, and the moment one of itseffects begins or becomes detectable. Above a certain amount of latencyintroduced by the network, some applications might become unusable:responsiveness of applications too slow, connections dropping. As aprior art example in FIG. 3, when a link (031) between two endpoints(026, 034) is established over the Internet and these two endpoints arenot located in the same continent, the overall performance of thecommunication is heavily degraded (packet loss, latency). In oneembodiment, performance degradation is solved by using regional hubs,where regional hubs are connected together using private high speed andlow latency circuits. As shown on FIG. 4, the spokes are connected tocloser hubs (041, 042, 043). The latter are connected together usingprivate high speed low latency links (040) featuring WAN optimizationtechniques. The WAN optimization engines are network appliances orsoftware that uses various WAN optimization algorithms that result in areduced amount of data to be sent across the network media to acceleratethe application performance and improve the user experience. Theseoptimization algorithms also called Wide Area Application Service (WAAS)(“WAN Optimization” and “WAAS” will be used interchangeably in thispatent) consists in, but in not limited to, Transport Flow Optimization(TFO), Data Redundancy Elimination (DRE), Adaptive PersistentSession-based Compression, Protocol Acceleration (application-layerread-ahead, operation prediction, message multiplexing, pipelining, andparallelization), Content Pre-Positioning, Meta-Caching. The WANoptimization process is summarized on FIG. 4 bis. WAN-optimized traffic(202) can only be “readable” between WAN optimization engines (201,203). When the traffic reaches (180) a point of presence (such as POP200 or POP 204), it is checked whether it is already WAN optimized(190). If the traffic is not WAN-optimized (190:no) and has to be sentto another point of presence (192), the local WAN optimization engine(201) will compress the traffic (191) using the WAN optimizationtechniques and sent it across the WAN network (202). The traffic willremain WAN-optimized until it reaches its final destination (181). Whenreceived by the final point of presence (181:yes, POP 204), the processof removing the WAN optimization is achieved (182). The traffic is thenhanded over in clear (183). Back to FIG. 3, the spoke based in Spain(024) is no longer connected to the hub (023) based in America using theInternet. In FIG. 4, the spoke (024) is connected to a regional hub(042) based in Europe. No longer aggregating spokes, the hub (023 inFIG. 3) becomes another spoke (023 in FIG. 4) in its regional networkand is connected to another aggregating hub (041) in America. The twohubs (041) and (042) are connected together using a private WANoptimized link (040). The spoke-to-spoke connectivity (047) is handledby the local Internet routing. No traffic transits via the ex-hub (023).Spoke-to-spoke connectivity can only occur within a same continent.Those skilled in the art should appreciate the gain of performance thatlocalized hubs bring to the network. The behavior of the Internet ispredictable when two endpoints communicate within the same continent.When transiting from one continent to another, the behavior andresulting performance cannot be easily anticipated. Using localized hubsbased in private points of presence (041, 042, 043) bring an end to endpredictable connectivity with dramatically reduced transit time (and asa consequence, resulting in a packet loss reduction and minimizednetwork latency). In order to the mGRE network to be establishedsuccessfully over the Internet, another protocol is used in order totranslate the public Internet IP addresses to the target IP addresseshandled by the spokes. As illustrated on FIG. 4 ter, a table (210) isbuilt by the hub (012) at the registration process (211) of the spokes(013, 014): all spoke IP addressing details are recorded at that time,both the internal LAN network details (IP subnets) (212, 213) and publicIP addresses. In the case of GRE, Next Hop Resolution Protocol (NHRP) isthe protocol used to translate the public IP addresses—also calledNon-broadcast Multiple Access Network (NBMA) addresses—to the target LANIP subnets (212, 213). An NBMA network is a network to which multiplecomputers and devices are attached, but data is transmitted directlyfrom one computer to another over virtual circuits like the GRE network.A hub acts as an NHRP server where the spoke are NHRP clients. A hub canalso act as an NHRP client towards other hubs. One successfulimplantation of mGRE over Internet using NHRP protocol is Cisco DynamicMultipoint VPN (DMVPN).

In order to build the mGRE network, the spokes have to have theendpoints IP routing information (IP routes to the NBMA IP addresses).In one embodiment, as shown on FIG. 5, the NBMA IP addresses of all theother endpoints are sent using remote agents based in a networkoperation centre (053) in order to optionally preserve the default IProute in each endpoint. The remote agents are automation daemons such asthose seen in Cisco VFrame solution or HP Opsware software. All the NBMAaddresses of the endpoints (012, 013, 014) are stored on a database(054) that the agents can use to generate the changes of the endpoints(012, 013, 014) routing table. The latter are sent to the endpointsusing a secure transport (049) like, but not limited to SNMP v3,SSH,SCP, SSL-based or TLS-based protocols. Those skilled in the art wouldappreciate that no default route is sent to the endpoints in order touse a default route within the private network (if needed). That is theultimate goal of that embodiment. Once the mGRE network is built, hubsand spokes are connected together. The resulting network is the NetworkAbstraction Layer (NAL).

The NAL may also be formed by a collection of network protocolsproviding the same subset of functionality provided by NHRP over mGRE orDMVPN as described earlier. The NAL can be formed by any protocols tobuild up the underlying network layer (NAL) as far as there is a directIP network link from one endpoint to another. These underlying could beLayer 2 Tunneling Protocol (L2TP) Point-to-Point Tunneling Protocol(PPTP), MultiProtocol Label Switching (MPLS), Overlay TransportVirtualization (OTV), Virtual Private LAN Switching (VPLS).

The Network Virtualization Layer (NVL) consists in adding an encryptionlayer on top of the NAL. IPSec (IP security) protocol is used toencode/decode the traffic. IPSec is a suite of protocols for securing IPcommunications by authenticating and/or encrypting each IP packet in adata stream. IPSec also includes protocols for cryptographic keyestablishment. Although IPSec provides a very high level of security,encryption and decryption processes are resource intensive. IPSecrequires cryptographic keys to be stored in memory. A cryptographic keyis required for each communication exchange with another endpoint. Eachendpoint has a key set and uses it to exchange data with anotherendpoint. In a network with many spokes, a large number ofspoke-to-spoke flows can end up in a resource starvation of theendpoints, degrading the network performance. In addition to that issue,the tunnel establishment takes time and is not compatible with timesensitive applications. In one embodiment, the NVL is based on the GDOIprotocol to overcome these two known issues: the GDOI protocol (GDOI)adds advanced endpoint resource management in a complex network topologyand removal of the tunnel establishment time. The virtual links betweenendpoints are instantly available. GDOI allows distributing the sameencryption key to every endpoint of the cloud as shown in FIG. 6. As aconsequence, only one key (062) is required for an endpoint (051) tocommunicate with any other endpoint (050, 052). All members get thatsame key (062) from a key server (KS) (060 or 061) and are called GroupMembers (GM). The KS (060, 061) is a key component of a GDOI domain(059). The KS are responsible for distributing and synchronizing thesame encryption key to all the members (050, 051, 052) of the domain(059). The encryption key is distribution mechanism (063) can beachieved using either unicast or multicast transport. The number of KSwithin a GDOI domain is not limited to one server: several synchronizedservers can deploy the same encryption key whilst sharing the workload.Before getting an encryption key from the KS, an endpoint needs toauthenticate to the KS. The authentication process can be achieved usingX.509 certificates or pre-shared (secret) keywords. The Virtual VPNsolution relies on certificates as the level of security is much higherthan secret keywords. As shown on FIG. 7, in one embodiment, thecertificate delivery process is performed by load-balanced subordinatecertification authorities (CA) (071, 072) to minimize the root CA (070)exposure and to add scalability to the network design. A certificationauthority (CA) is an entity which issues digital certificates for use byother authenticated parties. The root CA (070) is taken offline (073) tomitigate the risk of network attacks from the outside world. It willonly be used to renew or create certificates of the subordinatecertification authorities. All GMs (051, 052), including the KS (060)receive a signed certificate (074, 075, 076) from the subordinate CAs(071 or 072). The Simple Certificate Enrolment Protocol (SCEP) used todeliver the certificates (079). SCEP is a protocol using Hyper TextTransfer Protocol (HTTP) as the transport mechanism. It is thereforevery easy to deploy network load-balancers (NLB) (078) to load share thesubordinate CAs (071, 072). Network load-balancers (NLB) are networkequipments that share the network load to several network devices towhich it is connected to. All these network devices are seen as onelogical network device. The NLB (078) arbitrates and redirects thetraffic to the most available network device based upon relevantcriteria like, but not limited to, the current load, the response time.As a consequence, all the endpoints (051, 052, 060) configuration onlyrequire to point at the NLB (078) IP address rather than having tomanually be configured on which subordinate CA (071 or 072) eachendpoint should get their certificates. The NLB (078) also load-balancesthe HTTP servers that publish the Certificate Revocation Lists (CRL)(077). A CRL is a list of certificates serial numbers which have beenrevoked, are no longer valid, and should not be relied on by anyendpoint. Publishing CRLs is a way of ensuring validity of certificates.When the certificate enrolment is achieved successfully, each endpointhas a valid certificate (074, 075, 076) to claim an encryption key fromthe KS (060). Still part of the same embodiment, a certificate renewalprocess is set up to automatically occur when the certificate as aboutto expire. A reasonable amount of time like 10% of the certificate Ilifetime is given to ensure enough time is given to the certificaterenewal process so that no endpoints end up with no certificates.

Once the endpoint has received the X.509 certificate, it will connect tothe KS to get an encryption key. If the certificate is valid, theauthentication process is successful and the KS will deliver the currentencryption key along with all the other following keys. Likecertificates, each encryption key has a lifetime. When its lifetimeexpires, the encryption key is no longer usable. When the encryption keyis about to expire, a key encryption renewal process needs to occur.Again, a fair amount of time is given to the renewal process to avoid anencryption key starvation on the group members. The encryption keyrenewal process is identical as the key distribution process that hasbeen described in FIG. 6. When a group member (050, 051, 052) receives anew encryption key (062), the old key is discarded. The key (062)distribution process is achieved by one primary KS (060) and severalsecondary KS (061). The primary KS is responsible for the encryption keygeneration and synchronization of the encryption key with all thesecondary KS. Both primary and secondary KSs can distribute encryptionkeys to group members. This technique allows to spread the distributionworkload across all available KSs.

In a data center, in a normal situation, there are plenty of freeavailable resources in each performing device: available disk space,idle CPU time, or free memory. All these dedicated resources areinefficiently allocated because not shared with the other devices and asa result, are just wasted. For instance, an overwhelmed device out ofmemory could use some of the free memory space of the neighboringdevice. Virtualization is a device capability that solves theinefficiency of use of the available resource pool within a physicaldevice. For instance, it is unlikely to find a router with 99% ofcurrent CPU and memory use: firstly, because such a router will bereplaced very soon to avoid any service performance degradation andsecondly, because it will appear as a failure to size the routerspecifications accurately as the router is currently overwhelmed. Thatalso means that, in the opposite scenario, when that router is notrunning at 99%, there is a waste of available resource that could beuseful somewhere else. Virtualization addresses that resourcemanagement. A device capable of virtualization can be seen as aconsolidation of many virtual smaller devices, sharing the availablepool of resource of the physical device. The pool of resource, allocateddynamically, consists in, but is not limited to, CPU, memory (RAM orFlash), interfaces, I/O slots, disk space.

The additional advantage of virtualization is the simpler creation ofthese virtual devices: this creation is reduced to the remoteconfiguration of the physical device to enable a new virtual deviceassuming the wiring of the physical device is done accordingly inadvance. The wiring of the physical device has to be planned in such away that the incoming and outgoing connections to and from the physicaldevice are also virtualized to reduce the configuration of the physicaldevice interfaces to a simple set of commands sent remotely. Thoseskilled in the art would appreciate that 802.1q VLAN tagging describedunder the IEEE 802.1q standard is such a widely used technique to createmany virtual links under a common physical LAN connection.

A VPN aggregator endpoint, also called a VPN head-end (VHE), is theintelligence of a VPN network, in charge of, but not limited to, theendpoint registrations, the distribution of the network routing to allthe endpoints. In one embodiment, as shown in FIG. 8, one physical VHE(110) is aggregating traffic from different GDOI domains to act as manyvirtual VHEs. For instance, a GDOI domain (080, 090, 100) built on topof an mGRE network (NAL) (081, 091, 101). All the members of this GDOIdomain register (082, 092, 102) on the VHE (110), using their encryptionkey (112) securely received (111) by the KS (060). The traffic of allthese members is encrypted/decrypted with that same encryption key bythe VHE (110). The key is unique to the GDOI domain it is associated to.The VHE hands over (083, 093, 103) the traffic from the GDOI domain(080, 090, 100) to a dedicated network (084, 094, 104). Also, the samephysical VHE also handles other GDOI domains, using the differentencryption keys (112) accordingly: the different GDOI domains do notcommunicate with each other. Front Door Virtual Path Forwarding (fVRF)is one of the virtualization technique that can be used to achieve this.This is a virtualization technique. The VHE (110) is a physical endpointseparating the external side (001) with the internal side (043) of thenetwork. The latter reside in a point of presence. Several VHEs can bedeployed to ensure scalability and resiliency.

When virtualization is an advanced technique to aggregate severalcustomers onto same physical equipments, it is only enforceable on localequipments. Even when a customer is willing to improve their userexperience by converting their global VPN with unpredictable performance(as seen earlier when this global VPN includes inter-continental virtualtunnels over the public infrastructure) to regional VPNs, the latterneeds to be connected together to build the global network. In oneembodiment, a virtualized core stitches all regional VPNs together inorder to extend the customer reach seamlessly. A service provideroffering regional VPNs to his customers is able to build highlyperforming global VPN networks by getting regional VPNs connected toeach other instantly resulting in a significantly reduced time ofdeployment and reduced costs. As the core is virtualized, only onephysical infrastructure is required to transport all customers traffic.The virtualization techniques that can be used to build up the core are,but not limited to, Multiprotocol Label Switching (MPLS)-based networksincluding Layer 2 VPN (L2VPN) MPLS and Layer 3 VPN (L3VPN) MPLS, VirtualPrivate LAN Service (VPLS), Overlay Transport Virtualization (OTV),Frame-Relay, Encapsulating protocols like Generic Routing Encapsulation(GRE), Multipoint GRE (mGRE), 802.1q in 802.1q (Q-in-Q) 802.1adprotocol. As shown on FIG. 8 bis, two VHEs (114, 115), each of themlocated in distinct points of presence (141, 142) located in differentcontinent (145, 146) and a core network (143) getting them connectedtogether (144). Every VHE (114, 115) can potentially be attached to thecore (143) to allow a customer to build a global network. For eachnetwork created on a VHE, a virtual network is built up on the core tomake it instantly extendable to other regions. On the VHE side, as shownon FIG. 8, the VHE has connections to the Internet (001) and connectionsto the core network (043). The virtualization process happens on thecore side of the VHE. Each customer's NVL terminating on the VHE isbeing assigned a logical interface on the core side of the VHE,resulting in the entire customer's traffic to be handed over to theright customer's logical network in the core. For instance, the traffic(082) of GDOI domain (080) for a given customer, terminating on the VHE(110), will be handed over (083) to the right customer's logical network(084) on the core (043). On the core side, as shown on the FIG. 8 bis,when a VPN network is built on the VHE (114) to reach other VPN networkhosted on another VHE (115), the virtual network hand-off (083) iscreated on the VHE (114) internal physical interface (116). The latter(116) being connected on the core edge (147). The core edge (147) is anetwork equipment that is part of the core network (143) and responsiblefor managing the edge of the core network (143) or how the data flowsenter or exit the core network. As the traffic (084) reaches the coreedge (147), the traffic (086) then is sent over (144) the core (143) tothe right destination (146). The core edge (148) hands over the traffic(088) to the right virtual interface (089) of the other VHE (115) viathe physical interface (117). That demonstrates that two VPN networkscreated on two different VHEs (114, 115) can communicate with each otherusing the virtualized core network (143). In order to protect thecontrol plane traffic, all the endpoints should enforce network Qualityof Service (QoS). In routing, the control plane is the part of therouter architecture that is concerned with handling the information in a(possibly augmented) routing table that defines what to do with incomingnetwork traffic. Control plane logic also can define certain packets toget preferential treatment of certain packets for which a high qualityof service (QoS) is defined by such mechanisms as differentiatedservices. These packets that are critical are, for instance, keepalivepackets maintaining a communication channel between peers. Those skilledin the art should appreciate that, on a network where congestion doesnot occur, QoS will only be used to optimize the scheduling (queuing)process of the outgoing traffic of an endpoint. This paper describes howto build multiple virtualized GDOI domains on top of distinguished mGREnetworks over the Internet on mutualized physical platforms.

In one embodiment, the QoS engine running on each endpoint of thesenetworks is enforcing advanced traffic management to control andoptimized the data packets behavior over the last mile. The last mile isthe circuit directly connected to an endpoint. Most of the congestionhappens at that point. Once the traffic has reach the service provider'score network (at the other end of that circuit), there is unlikely tohave bandwidth starvation occurring. As illustrated on FIG. 9, thetraffic is classified in four categories: control plane (121),management plane (122), critical applications (123) and best effort(124). Part of this embodiment, the traffic is released from theendpoint in the following order: first the control plane traffic, thenthe management plane traffic, then the critical applications trafficthen the best effort traffic. The control plane comprises of all thetraffic that is necessary to keep the network up and running for boththe NAL and NVL. Some examples of control plane traffic includes, but itnot limited to, the GDOI key exchange traffic, the Internet Key Exchange(IKE) key exchange, the SCEP traffic for the X.509 certificate enrolmentprocess, the IKE Dead Peer Detection (DPD) keepalive packets, theBidirectional Forwarding Detection (BFD) traffic, the dynamic routingprotocol exchanges, the IP SLA agent/responder traffic, the NHRPregistration/notification/update traffic, the CRL publication traffic,any signaling traffic (in multimedia data flows like voice or video).The control plane traffic will be treated as the most important traffic,therefore processed accordingly by an endpoint: this traffic will beunleashed to the endpoint network interface transmit ring (125) first tobe sent out to the network (127) before any other traffic. The trafficmay be sent using connection (126). Those skilled in the art shouldappreciate that some of the control plane traffic will be ciphered. Whenavailable, a data packet pre-classification feature will be enabled onall endpoints where encryption occurs in order to their QoS engine toprocess the packets accordingly. The management plane comprises of allthe traffic used to manage the endpoints. Some examples of managementplane traffic includes, but it not limited to, Secure Shell (SSH),Secure Socket Layer (SSL)-based, Transport Layer Security (TLS)-based orTelnet protocol, Simple Network Management Protocol (SNMP) traffic, someof the Trivial File Transfer Protocol (TFTP), File Transfer Protocol(FTP) or Hyper Text Transfer Protocol (HTTP) traffics for firmware orcode upgrades. The critical applications traffic comprises of all thenetwork flow coming from important identified corporate applications.Some examples of critical applications traffic includes, but it notlimited to, Enterprise Resource Planning (ERP) traffic, provisioningprocess traffic. All the traffic that has not been classified undereither control plane or management plane or critical applications willbe treated in a best effort manner.

On a network interface, scheduling (queuing) occurs. Each networkinterface queues up a certain amount of traffic before releasing it ontothe network media. The controlling process of these queues is called“transmit ring”. Once the transmit ring is full, the network packets inthe buffers are sent onto the network. When the transmit ring waits tobe filled up, some critical network packets might be delayed, affectingthe network performance or worse, compromising the network stability. Inone embodiment, the transmit ring queue length is tweaked in order toreduce the delay before network packets are released on the networkmedia. For instance, on DSL ports, the default transmit ring queuelength is set to 64 packets on most endpoints. On Ethernet interfaces,the default transmit ring queue length is set to 128 packets. Part ofthis embodiment, the transmit ring queue length is reduced to a verysmall value (below 5 packets). Those skilled in the art shouldappreciate that reducing the transmit ring queue size also overcomes theperformance degradation introduced by oversubscription of DSL accesses.

In one embodiment, the provisioning process of endpoints is achieved inan unattended manner.

The provisioning process consists in two tasks: first task is toconfigure the VHE and the second task is to provision and configure theendpoints that will connect and register to the VHE. The VHE asdescribed earlier in FIG. 8 is a very powerful endpoint handlingdifferent NVLs and NALs. All the network configuration aspect is basedusing virtualization capabilities of the VHE. All the configurationprocess consists in creating virtual instances of the available physicalresource pool of the VHE. There is no extra physical wiring exercise tobe done as all the wiring has been done in advanced. As no physicalwiring is required, the provisioning process is reduced to send theconfiguration details to the VHE remotely. As shown on FIG. 10, facingthe VHE, the endpoints that are installed on the other side of thenetwork (051, 052), most likely in the customer's premises, have aprovisioning process reduced to the minimum. The endpoints are deployedwith a very simple configuration, also called bootstrap configuration. Abootstrap configuration is the minimum configuration required to get theendpoint (051, 052) connected to the network, in that case, the Internet(001). With this bootstrap configuration, the endpoint (051, 052) isremotely accessible by the automation engine (130, 131) that are incharge of “injecting” the complete device configuration. Then, theconfigurations (132) for both the VHE (012) and endpoints (051, 052) aregenerated and sent remotely (135) by automation engine (130, 131)sitting in the network operation centre (NOC) (053) as illustrated onFIG. 10. The automation engine comprise of a configuration engine (130)storing all endpoints details in a repository, an orchestratorgenerating the configurations (132) of the endpoints and deliveryengines (131) sending configurations to endpoints (051, 052). Thecomplete explanation of how the automation engine works is explained inProvisional Patent Application Ser. No. 61/121,127 filed Dec. 9, 2008,entitled “Remote VPN Device Automation Engine, incorporated herein byreference. Eventually, the remote configuration process for the VHEs canbe achieved using an out-of-band management network (136) to increasethe security of the configuration process. An out-of-band managementnetwork acts like a back door that can be used in case the in-bandnetwork fails. The FIG. 11 describes the remote configuration process ofboth VHEs and endpoints performed by the automation engine describedearlier. All the NBMA IP addresses of the other existing endpoints ofthe network are sent (150) in the IP routing tables of the VHEs andendpoints to configure. An IP routing table is a database in a routerthat stores the routes to network destinations. These routes arerequired for the network traffic to reach a particular destination. Whena router does not have sufficient routing information on a givendestination, it relies on a default route, if present, that should leadto that particular unknown destination. The remote automation enginewill never send a default route to the endpoints as a default routemight need to be saved for some other applications and that there can beonly one default route on an endpoint. The management configurationdetails are sent (151). They comprise of, but are not limited to, theadministrator credentials (usernames, password and access level), views,logging details, reporting details. These are used for getting access tothe endpoint and the latter to report all its activities. The interfaceports configuration is added (152). This comprises of, but is notlimited to, the interface names, interface speed, interface specificdetails (PVC, Tag, Label, MTU, TCP MSS . . . ), security settings(firewall, filters, ACLs) and interface hardware settings. These areused for connecting and securing the endpoint. The QoS engineconfiguration is added to the configuration (153). This comprises of,but is not limited to, the classification of the network packets, thetransmit ring queue length, the network policer engine details, thenetwork shaping engine details. These are used for improving theendpoint performance and deal with network congestion and bottlenecks.The X.509 certification enrolment configuration details are added to theconfiguration (154). This comprises of, but is not limited to, the rootcertificate string, the trustpoint IP address, the certificate requestdetails (CN, O, OU, E), the lifetime of the certificate. These are usedfor authenticating the endpoints and prevent man-in-the-middle networkattacks. The Network Virtualization Layer (NVL) Key Server (KS) detailsare added (155). This comprises of, but is not limited to, the KS IPaddresses, the GDOI identity number. These are used for the endpoints toretrieve the encryption keys and get access to the right GDOI domain.Some extra KS configuration details are added the first time the KS iscreated for a new network. This comprises of, but is not limited to,rekey IP addresses, rekey retransmit delay, the GM authorization ACL,the IPSec security association details, the anti-replay strategy. Theseare used for securing the GDOI domain. The Network Abstraction Layer(NAL) access details are added (156). This comprises of, but is notlimited to, the NHRP network details (ID, key, static entries, shortcutswitching activation, routing engine changes), the GRE source interfacesor IP addresses, the GRE destination IP addresses, the QoS engine touse. These are used for the endpoints to get access to the NAL. TheNetwork Virtualization Layer (NVL) network configuration is added (157).This comprises of, but is not limited to, the GDOI identity number, theKS IP addresses, the GDOI encryption details (ACLs, IPSec transformset). These are used for the endpoints to connect to their associatedGDOI domain. The dynamic routing protocol is added to the configuration(158). This comprises of, but is not limited to, the routing protocolconfiguration used over the network, the interface costs, theBidirectional Forwarding Detection (BFD) peers, the IP SLA agents. Theseare used for the endpoints to retrieve the dynamic routing information.Finally, the LAN interface configuration is sent. This comprises of, butis not limited to, the interface speeds, the interface cost, theinterface specific settings (MTU, TCP MSS), the high availabilityprotocol settings (VRRP, HSRP, GLBP). These are used for configuring theLAN-facing interface configuration (159) of the endpoints.

Using the teachings outlined in the above written description includingits figures and also with the knowledge of the commercial hardware suedin the implementation of the network, one of skilled in the art canwrite scripting router configurations and configure the hardware andsoftware as required to implement our solution.

Alternative Implementations

The processes of this patent can be implemented in a number of ways. Thefollowing are some, but not all, of the ways in which such processes canbe carried out.

The processes can be carried out by a person keying instructions into acomputer to provision a communication system to operate as disclosedherein. They can also be carried out by a system itself, and also by theinteraction of a server and a client, or the interaction of endpointspeered with each other exchanging data packets. There is any number ofsuch means for carrying out the processes.

Further, the processes can be implemented by an article of manufacture,such as a storage medium with the processes embedded thereon or thereinin the form of computer instructions or otherwise. Such medium could be,without limitation, optical storage such as CD, DVD, Blu-Ray, or othersuch optical storage. A medium could also be flash memory-based storage.Such medium could contain a copy of programming instructions on internalcomputer memory during a download process or during an upload process.Further, the storage medium could be the memory of an integratedcircuit, such memory having one or more of such processes stored thereonor therein.

In the foregoing specification, specific embodiments of the presentinvention have been described. However, one of the ordinary skills inthe art appreciates that various modifications and changes can be madewithout departing from the scope of the present invention as set forthin the claims below. Accordingly, the specification and figures are tobe regarded in an illustrative rather than a restrictive sense, and allsuch modifications are intended to be included within the scope ofpresent invention. The benefits, advantages, solutions to problems, andany element(s) that may cause any benefit, advantage or solution tooccur or become more pronounced are not to b construed as a critical,required, or essential features or elements of any or all the claims.The invention is defined solely by the appended claims including anyamendments made during the pendency of this application and allequivalents of those claims as issues.

What is claimed is:
 1. A process of communication, comprising a networkabstraction layer (NAL) built on a public Internet; and a networkvirtualization layer (NVL) built on the NAL.